Selection of reference sites is a critical component in the process of developing and applying biological indicators of ecological condition. Some evidence suggests that despite its importance the rules by which reference sites are selected have not always been evaluated critically to assure that the sites represent least-disturbed conditions. We present a comparison of physical and chemical disturbance measures and biotic indices at handpicked reference sites provided by resource agencies and at sites selected by a probability design from a 12-state survey of western streams and rivers. In most cases, the distributions of disturbance measures and biotic index scores were essentially the same for both types of sites; that is, only a subset of the handpicked reference sites represented least-disturbed conditions. We recommend that all agencies that use reference sites critically review those sites with a set of explicit criteria, using field-collected data as well as mapped information.